fix(sdk): align eval config providers surface - #1722
Merged
Conversation
Deploying agentv with
|
| Latest commit: |
1ab307b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4730d8a3.agentv.pages.dev |
| Branch Preview URL: | https://av-icbt-sdk-provider-surface.agentv.pages.dev |
christso
force-pushed
the
av-icbt-sdk-provider-surface
branch
from
July 7, 2026 08:16
89a2524 to
40eb768
Compare
christso
force-pushed
the
av-icbt-sdk-provider-surface
branch
from
July 7, 2026 08:18
40eb768 to
1ab307b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TypeScript eval authoring now follows the same Promptfoo-shaped provider surface as YAML. SDK users can author
providersentries withid/label, selectdefaults.provideranddefaults.grader, and set grader providers throughdefaultTest.options.provider,tests[].options.provider, or assertion-levelproviderwithout falling back to top-leveltarget,targets, orgraders.The TS eval loader no longer rewrites YAML-aligned
targetintoproviders; stale target-era eval files now fail with provider migration guidance.options.provideris also carried into LLM-backed assertions as the inherited grader provider, while assertion-levelproviderstill wins.The remaining target-named SDK API is
createTargetClient()and related target-proxy types. Those are documented as runtime-only script-grader proxy APIs, not eval authoring syntax.Validation
bun test packages/sdk/test/eval-authoring.test.tsbun test packages/core/test/evaluation/loaders/ts-eval-loader.test.ts packages/core/test/evaluation/loaders/grader-parser.test.tsbun --filter @agentv/core typecheckbun --filter @agentv/sdk typecheckbunx biome check ...touched files...bun --filter @agentv/sdk testbun --filter @agentv/sdk buildLive dogfood was attempted after copying
.envfrom the primary checkout. It is blocked locally: eval-file inline providers are still ignored by the current CLI provider resolution path covered by #1718, catalogopenaistarts but fails withpi-ai call failed: Connection error, and cataloglocal-openairequires missingLOCAL_OPENAI_PROXY_API_KEY.Related
Related: av-icbt
Related: #1718, #1719